Medical image processing device, medical observation system, and image processing method

ABSTRACT

A medical image processing device includes an image processing unit configured to perform image processing, based on image data generated by imaging reflected light of first visible light emitted to an object and fluorescence when a light source device simultaneously emits the first visible light and excitation light that excites a fluorescent substance to emit the fluorescence. The image processing unit is configured to generate an interpolation pixel value corresponding to a component of second visible light in a wavelength band different from a wavelength band of the first visible light, based on a first pixel value and output from a pixel receiving the reflected light of the first visible light emitted to the object, generate a background image based on the first pixel value and the interpolation pixel value, and generate a fluorescence image based on a second pixel value included in the image data and output from a pixel receiving the fluorescence.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Japanese Application No.2020-049175, filed on Mar. 19, 2020, the contents of which areincorporated by reference herein in its entirety.

BACKGROUND

The present disclosure relates to a medical image processing device, amedical observation system, and an image processing method.

In surgical microscopes, there is known a technique in which red light,green light, and blue light are guided to three image sensors via adichroic beam splitter, and infrared light is guided to one of the threeimage sensors to observe a fluorescence image and a visible light image(see, e.g., JP 5646844 B2).

SUMMARY

In JP 5646844 B2 described above, the fluorescence image and the visiblelight image are observed using three image sensors, and thus, there is aproblem that reduction in size of the device is made difficult.

There is a need for a medical image processing device, a medicalobservation system, and an image processing method that are able toreduce a size of a device.

According to one aspect of the present disclosure, there is provided amedical image processing device including an image processing unitconfigured to perform image processing, based on image data generated byimaging reflected light of first visible light emitted to an object andfluorescence, by a medical imaging device, when a light source devicesimultaneously emits, to the object, the first visible light andexcitation light that excites a fluorescent substance to emit thefluorescence, wherein the image processing unit is configured togenerate an interpolation pixel value corresponding to a component ofsecond visible light in a band different from that of the first visiblelight, based on a first pixel value included in the image data andoutput from a pixel receiving the reflected light of the first visiblelight emitted to the object, generate a background image based on thefirst pixel value and the interpolation pixel value, and generate afluorescence image based on a second pixel value included in the imagedata and output from a pixel receiving the fluorescence.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the overall configuration of a medicalobservation system according to a first embodiment;

FIG. 2 is a block diagram illustrating a functional configuration of themedical observation system according to the first embodiment;

FIG. 3 is a diagram schematically illustrating a configuration of apixel portion according to the first embodiment;

FIG. 4 is a diagram schematically illustrating a configuration of acolor filter according to the first embodiment;

FIG. 5 is a diagram schematically illustrating the sensitivity of animage sensor in each wavelength band after passing through the colorfilter according to the first embodiment;

FIG. 6 is a flowchart illustrating the outline of a process performed bythe medical observation system according to the first embodiment;

FIG. 7 is a diagram schematically illustrating the outline of a processin a fluorescence observation mode performed by the medical observationsystem according to the first embodiment;

FIG. 8A is a graph illustrating a correlation between pixels located atthe same positions of a red image, a green image, and a blue image thatare obtained by separating a medical image of a surgical site(biological tissue) of a subject;

FIG. 8B is a graph illustrating a correlation between pixels located atthe same positions of the red image, the green image, and the blue imagethat are obtained by separating the medical image of the surgical site(biological tissue) of the subject;

FIG. 8C is a graph illustrating a correlation between pixels located atthe same positions of the red image, the green image, and the blue imagethat are obtained by separating the medical image of the surgical site(biological tissue) of the subject;

FIG. 9A is a graph illustrating a correlation between pixels located atthe same positions of a red image, a green image, and a blue image thatare obtained by separating another medical image of a surgical site(biological tissue) of a subject;

FIG. 9B is a graph illustrating a correlation between pixels located atthe same positions of the red image, the green image, and the blue imagethat are obtained by separating the another medical image of thesurgical site (biological tissue) of the subject;

FIG. 9C is a graph illustrating a correlation between pixels located atthe same positions of the red image, the green image, and the blue imagethat are obtained by separating the another medical image of thesurgical site (biological tissue) of the subject;

FIG. 10 is a table illustrating a correlation between pixels located atthe same positions of red images, green images, and blue images that areobtained by separating a plurality of medical images of a surgical site(biological tissue) of a subject;

FIG. 11 is a diagram schematically illustrating a generation method forgenerating a fluorescence image by a first generation unit according tothe first embodiment;

FIG. 12 is a schematic diagram illustrating another example of thegeneration method for generating a fluorescence image by the firstgeneration unit according to the first embodiment;

FIG. 13 is a diagram schematically illustrating a generation method forgenerating a background image by a second generation unit according tothe first embodiment;

FIG. 14 is a conceptual diagram illustrating a configuration of animaging unit according to a modification of the first embodiment;

FIG. 15 is a diagram illustrating a schematic configuration of a medicalobservation system according to a second embodiment;

FIG. 16 is a block diagram illustrating a functional configuration of amain portion of the medical observation system according to the secondembodiment; and

FIG. 17 is a diagram illustrating a schematic configuration of a medicalobservation system according to a third embodiment.

DETAILED DESCRIPTION

Modes for carrying out the present disclosure (hereinafter, referred toas “embodiments”) will be described below in detail with reference tothe drawings. Note that the present disclosure is not limited to thefollowing embodiments. In addition, the drawings referred to in thefollowing descriptions are merely schematically illustrated in shape,size, and positional relationship so as to understand the contents ofthe present disclosure. In other words, the present disclosure is notlimited only to the shapes, sizes, and positional relationshipsexemplified in the drawings.

First Embodiment

Schematic Configuration of Medical Observation System

FIG. 1 is a diagram illustrating the overall configuration of a medicalobservation system according to a first embodiment. A medicalobservation system 1 illustrated in FIG. 1 includes a medicalobservation device 2, a light source device 3, a display device 8, and acontrol device 9, the medical observation device 2 being configured tofunction as a microscope to magnify a minute portion of an object to beobserved, for observation, the light source device 3 being configured tosupply illumination light to the observation device 2 via a light guide4 including optical fiber or the like, the display device 8 beingconfigured to display an image based on image data captured by theobservation device 2, the control device 9 being configured tointegrally control the operation of the medical observation system 1.

Schematic Configuration of Observation Device

First, a schematic configuration of the observation device 2 will bedescribed. The observation device 2 includes a microscope unit 5, asupport unit 6, and a base portion 7. The microscope unit 5 isconfigured to observe a minute portion of an object to be observed, thesupport unit 6 is connected to a base end portion of the microscope unit5 and configured to rotatably support the microscope unit 5, and thebase portion 7 is configured to rotatably hold a base end portion of thesupport unit 6 and to be movable on a floor surface.

The microscope unit 5 has a columnar appearance and internally includesan optical system, an image sensor (not illustrated), and a lightemitting unit (not illustrated), the optical system having a zoom and afocus function, the image sensor being configured to receive lightfocused into an image of an object by the optical system, performphotoelectric conversion, and generate image data, the light emittingunit being configured to emit illumination light to an object to beobserved. Furthermore, the microscope unit 5 has a side surface on whichvarious switches are provided, the various switches constituting aninput unit 24 receiving input of operation instruction for theobservation device 2. The microscope unit 5 has an opening surface at alower end and the opening surface is provided with a cover glass (notillustrated) that protects the optical system and the like positionedtherein. A user such as an operator is allowed to move the microscopeunit 5, change the angle of the microscope unit 5, change the modes ofthe observation device 2, or perform zoom or focus operation, whileoperating the various switches with the microscope unit 5 held. Theshape of the microscope unit 5 is not limited to the cylindrical shapebut may be, for example, a polygonal cylindrical shape.

Under the control by the control device 9, the light source device 3supplies, to the observation device 2 via the light guide 4,illumination light of at least one of infrared light and white lightincluding light in a red wavelength band, light in a green wavelengthband, and light in a blue wavelength band. The light source device 3includes a discharge lamp such as a xenon lamp or metal halide lamp, asolid-state light emitting device such as a light emitting diode (LED)or a laser diode (LD), or a light emitting member such as a halogenlamp.

The display device 8 displays an image to be displayed that is generatedby the control device 9 or various information about the medicalobservation system. The display device 8 includes liquid crystal,organic electro luminescence (EL), or the like. The display device 8displays a 2D image or 3D image.

The control device 9 integrally controls the respective units of themedical observation system 1. The control device 9 is achieved by usinga memory and a general-purpose processor such as a central processingunit (CPU) or a processor including hardware such as various arithmeticcircuits performing specific functions, such as an application specificintegrated circuit (ASIC) or graphics processing unit (GPU).Furthermore, the control device 9 may include a field programmable gatearray (FPGA: not illustrated) that is a kind of programmable integratedcircuit. Note that when the FPGA is included, a memory for storingconfiguration data may be provided so that the FPGA as the programmableintegrated circuit is configured based on the configuration data readfrom the memory. The configuration of the control device 9 will bedescribed in detail later.

Functional Configuration of Medical Observation System

Next, the functional configuration of the medical observation system 1will be described. FIG. 2 is a block diagram illustrating the functionalconfiguration of the medical observation system 1.

Functional Configuration of Observation Device

First, the functional configuration of the observation device 2 will bedescribed.

The observation device 2 includes the microscope unit 5, a detectionunit 23, the input unit 24, and a first control unit 25.

The microscope unit 5 includes an imaging unit 21 and a light emittingunit 22, the imaging unit 21 being configured to generate image data bymagnifying an image of the object to be observed that is an observationtarget, the light emitting unit 22 being configured to irradiate theobject to be observed with illumination light supplied from the lightsource device 3.

The imaging unit 21 includes an optical system 211, an image sensor 212,and a cut filter 213. Note that the imaging unit 21 functions as amedical imaging device according to the first embodiment.

The optical system 211 has a zoom and a focus function and forms animage of the object on a light receiving surface of the image sensor 212via the cut filter 213. The optical system 211 is achieved by using oneor a plurality of lenses, a motor configured to move the lenses along anoptical path L1, and the like.

The image sensor 212 receives light focused into an image of an objectby the optical system 211 via the cut filter 213, performs photoelectricconversion, and generates image data (RAW data). The image sensor 212 isachieved by using an image sensor, such as a charge coupled device (CCD)or complementary metal oxide semiconductor (CMOS). The image sensor 212includes a pixel portion 212 a and a color filter 212 b.

FIG. 3 is a diagram schematically illustrating a configuration of thepixel portion 212 a. As illustrated in FIG. 3, in the pixel portion 212a, a plurality of pixels P_(n,m) (n=an integer of 1 or more, m=aninteger of 1 or more), such as photodiodes, accumulating electric chargeaccording to an amount of light is arranged in a two-dimensional matrix.Under the control by the first control unit 25, the pixel portion 212 areads an image signal as image data from a pixel P_(n,m) in a read areaset as a read target, of the plurality of pixels P_(n,m), and outputsthe image data to the control device 9. Specifically, the image datagenerated by the pixel portion 212 a is transmitted to the controldevice 9 via a transmission cable. Note that the image data generated bythe pixel portion 212 a may be subjected to E/O conversion so as to betransmitted to the control device 9 by an optical signal.

FIG. 4 is a diagram schematically illustrating a configuration of thecolor filter 212 b. The color filter 212 b illustrated in FIG. 4includes a Bayer array having 2×2 filters as one unit. The color filter212 b includes a filter R configured to transmit light in a redwavelength band, two filters G (filter Gr, filter Gb) configured totransmit light in a green wavelength band, and a filter B configured totransmit light in a blue wavelength band.

In the following description, a pixel P_(n,m) having a light receivingsurface on which the filter R is arranged is referred to as an R pixel,a pixel P_(n,m+1) having a light receiving surface on which the filterGr is arranged is referred to as a Gr pixel, a pixel P_(n,m+1) having alight receiving surface on which the filter Gb is arranged is referredto as a Gb pixel (hereinafter, the Gr pixel and the Gb pixel arecollectively referred to as a G pixel), and pixels P_(n+1,m+1) having alight receiving surface on which the filter B is arranged is referred toas a B pixel. Furthermore, in the first embodiment, the filter Rfunctions as a first filter configured to transmit light (first visiblelight) in a red wavelength band and fluorescence, the filter B functionsas a second filter configured to transmit light (second visible light)in a blue wavelength band and fluorescence, and the filter G functionsas a third filter configured to transmit light (third visible light) ina green wavelength band and fluorescence. In other words, a pixel valueof the R pixel includes components of reflected red light andfluorescence, a pixel value of the G pixel (Gr pixel, Gb pixel) includescomponents of reflected green light and fluorescence, and a pixel valueof the B pixel includes components of reflected blue light andfluorescence.

The cut filter 213 is arranged on the optical path L1 between theoptical system 211 and the image sensor 212. The cut filter 213 blockslight having a wavelength component (e.g., 740±10 nm) of excitationlight included in an image of the object formed by the optical system211 and transmits light having a wavelength component of light otherthan the excitation light.

Here, a spectral characteristic of each pixel will be described. FIG. 5is a diagram schematically illustrating the sensitivity of the imagesensor in each wavelength band after passing through the color filter.In FIG. 5, the horizontal axis represents wavelength (nm) and thevertical axis represents spectral characteristic. Furthermore, in FIG.5, a curve L_(B) represents a spectral characteristic of the B pixel, acurve L_(G) represents a spectral characteristic of the G pixel, a curveL_(R) represents a spectral characteristic of the R pixel, and astraight line L_(IR) represents a wavelength band of fluorescencegenerated by irradiating a fluorescent substance with excitation light.A curve L_(CUT) represents a transmission characteristic of the cutfilter 213.

As represented by the curve L_(B) and the straight line L_(IR) of FIG.5, the B pixel is sensitive to light in the blue wavelength band (435 nmto 480 nm) (hereinafter, simply referred to as “blue light”) and issensitive to fluorescence in a wavelength band (830±10 nm) generated byirradiating the fluorescent substance with excitation light.Furthermore, as represented by the curve line L_(G) and the straightline L_(IR) of FIG. 5, the G pixel (Gr pixel and Gb pixel) is sensitiveto light in the green wavelength band (500 nm to 560 nm) (hereinafter,simply referred to as “green light”) and is sensitive to fluorescence inthe wavelength band (830±10 nm) generated by irradiating the fluorescentsubstance with excitation light. Furthermore, as represented by thecurve line L_(R) and the straight line L_(IR) of FIG. 5, the R pixel issensitive to light in the red wavelength band (610 nm to 750 nm)(hereinafter, simply referred to as “red light”) and is sensitive tofluorescence in the wavelength band (830±10 nm) generated by irradiatingthe fluorescent substance with excitation light.

The light emitting unit 22 includes an illumination optical systemconfigured by using one or more lenses. The light emitting unit 22 emitsillumination light in the same direction as an imaging direction of theimaging unit 21, the illumination light being at least one of whitelight supplied from the light source device 3 via the light guide 4,light in a red wavelength band, light in a green wavelength band, lightin a blue wavelength band, and infrared light. Note that the lightemitting unit 22 may be provided with a light emitting diode (LED), alaser light source, or the like at the microscope unit 5 to omit opticaltransmission via the light guide or the like.

The detection unit 23 sequentially detects status information about theobservation device 2. The status information about the observationdevice 2 includes information about position, focus, and zoominformation about the imaging unit 21. The detection unit 23 includesvarious sensors to detect such information.

The input unit 24 receives input of an operation instruction to theimaging unit 21. The input unit 24 includes a focus switch and a zoomswitch each configured to receive an input of an instruction for focusor zoom operation in the imaging unit 21, an electric scrolling modeswitch configured to receive an input of an instruction for an electricscrolling mode, and a mode changeover switch configured to receive inputof instruction for changing an observation mode of the medicalobservation system 1. As illustrated in FIG. 1, various switches,buttons, and the like constituting the input unit 24 are provided on theside surface of the microscope unit 5.

The first control unit 25 controls the operation of the imaging unit 21in response to an operation instruction received by the input unit 24 oran operation instruction input from the control device 9 which isdescribed later. Furthermore, the first control unit 25 integrallycontrols the observation device 2 in cooperation with a second controlunit 94 of the control device 9 which is described later. The firstcontrol unit 25 includes a memory, and a processor such as a CPU, FPGA,or ASIC.

Configuration of Light Source Device

A configuration of the light source device 3 will be described next.

The light source device 3 includes a first light source unit 31, asecond light source unit 32, a third light source unit 33, and a fourthlight source unit 34.

Under the control by the control device 9, the first light source unit31 supplies red light to the light emitting unit 22 of the observationdevice 2 via the light guide 4. The first light source unit 31 isachieved by using a red LED or the like.

Under the control by the control device 9, the second light source unit32 supplies green light to the light emitting unit 22 of the observationdevice 2 via the light guide 4. The second light source unit 32 isachieved by using a green LED or the like.

Under the control by the control device 9, the third light source unit33 supplies blue light to the light emitting unit 22 of the observationdevice 2 via the light guide 4. The third light source unit 33 isachieved by using a blue LED or the like.

The fourth light source unit 34 supplies infrared light exciting thefluorescent substance to the light emitting unit 22 of the observationdevice 2 via the light guide 4. Under the control by the control device9, the fourth light source unit 34 supplies infrared light (in awavelength band of 740±10 nm) functioning as excitation light excitingthe fluorescent substance. The second light source unit 32 includes asemiconductor laser device configured to be able to emit infrared light(700 to 1000 nm) used for indocyanine green (ICG) observation, a filterconfigured to transmit only a predetermined wavelength band (awavelength band of 740±10 nm), and the like. Note that in the following,infrared light is described, but the excitation light is not limited tothis, and, for example, light (a wavelength band of 415±10 nm) used forphoto dynamic diagnosis (PDD) observation of fluorescence of aphotosensitive substance, such as hematoporphyrin derivative,accumulated in tumor tissue in advance may be employed, and light (awavelength band of 390 to 470 nm+a wavelength band of 540 to 560 nm)used for auto fluorescence imaging (AFI) for observation of autofluorescence from a fluorescent substance such as collagen may beemployed.

Configuration of Control Device

Next, the functional configuration of the control device 9 will bedescribed.

The control device 9 includes an image processing unit 91, an input unit92, a recording unit 93, and the second control unit 94.

The image processing unit 91 performs various image processing on imagedata transmitted from the observation device 2 to generate an image tobe displayed (video data) that is displayed by the display device 8.Here, examples of the image processing include various image processingand the like, such as color correction, color enhancement, and contourenhancement. Furthermore, the image processing unit 91 generates aninterpolation pixel value based on a first pixel value included in imagedata transmitted from the observation device 2, generates a backgroundimage based on the first pixel value and the interpolation pixel value,and generates a fluorescence image based on a second pixel valueincluded in the image data transmitted from the observation device 2.The interpolation pixel value corresponds to a component of the secondvisible light (blue light) that is in a band different from that of thefirst visible light (one of red light and green light), the first pixelvalue is output from a pixel receiving the first visible light (one ofred light and green light), and the second pixel value is output from apixel receiving fluorescence. The image processing unit 91 includes amemory and a processor such as a graphics processing unit (GPU), ASIC,or FPGA. The image processing unit 91 includes at least a subtractionunit 911, a first generation unit 912, a second generation unit 913, athird generation unit 914, and a combining unit 915.

The subtraction unit 911 subtracts the second pixel value from the firstor third pixel value included in image data input from the imaging unit21, and outputs a result of the subtraction to the second generationunit 913. The second pixel value is output from a pixel (B pixel) onwhich the second filter (filter B) is arranged, and the first or thirdpixel value is output from a pixel (R pixel or G pixel) on which thefirst or third filter (filter R or filter G) is arranged. Specifically,the subtraction unit 911 may divide the spectral sensitivity of thefluorescence wavelength of the first or third filter, by the spectralsensitivity of the fluorescence wavelength of the second filter toobtain a divided value, multiply the divided value by the second pixelvalue to obtain a multiplication result, subtract the multiplicationresult from the first or third pixel value to obtain a subtractionresult, and output the subtraction result to the second generation unit913. Note that the calculation method by the subtraction unit 911 willbe described later.

Under the control by the second control unit 94, the first generationunit 912 generates a fluorescence image based on the second pixel valueincluded in the image data input from the imaging unit 21 and outputsthe fluorescence image to the combining unit 915. The second pixel valueis output from a pixel (B pixel) on which the second filter (filter B)is arranged. Specifically, the first generation unit 912 generates thefluorescence image by interpolating the pixel values of the R pixel andthe G pixel, based on the pixel value of the B pixel included in theimage data, and outputs the fluorescence image to the combining unit915. Furthermore, the first generation unit 912 performs colorizationfor the fluorescence image. Specifically, the first generation unit 912colorizes the fluorescence image by a tone conversion process or thelike based on the brightness value of the fluorescence image, andoutputs the colored fluorescence image to the combining unit 915. Forexample, the first generation unit 912 performs the colorization tocolor a fluorescent area green based on the brightness value of thefluorescence image. Note that the first generation unit 912 may set acolorization color to the fluorescence image, based on a designationsignal input from the input unit 92 via the second control unit 94 tospecify the color of the fluorescent area of the fluorescence image.

Under the control by the second control unit 94, the second generationunit 913 generates a first background image, based on the first or thirdpixel value included in the image data input from the imaging unit 21and outputs the first background image to the third generation unit 914.The first or third pixel value is output from a pixel (R pixel or Gpixel) on which the first or third filter (filter R or filter G) isarranged. Specifically, the second generation unit 913 generates thefirst background image based on a result of the input from thesubtraction unit 911. Furthermore, the second generation unit 913generates the interpolation pixel value based on the first or thirdpixel value included in the image data input from the imaging unit 21and outputs the interpolation pixel value to the third generation unit914. The first or third pixel value is output from a pixel (R pixel or Gpixel) on which the first or third filter (filter R or filter G) isarranged, and the interpolation pixel value interpolates a pixel valuecorresponding to a component of one of red, green, and blue light thatis not included in the visible light.

The third generation unit 914 generates a second background image basedon the interpolation pixel value and the first or third pixel value (thefirst background image) that are input from the second generation unit913 and outputs the second background image to the combining unit 915.Furthermore, the third generation unit 914 may perform binarization forthe second background image and output the second background image tothe combining unit 915. For example, the third generation unit 914 mayperform a saturation reduction process to reduce the saturation of thebackground image and output the binarized second background image to thecombining unit 915.

The combining unit 915 generates a composite image in which thefluorescence image input from the first generation unit 912 and thesecond background image input from the third generation unit 914 arecombined, and outputs the composite image to the display device 8.Specifically, the combining unit 915 generates the composite image bycombining the fluorescence image with the second background image at apredetermined ratio (e.g., 1:1).

The input unit 92 includes a user interface such as a keyboard, mouse,touch panel, and foot switch, and receives input of various information.

The recording unit 93 is constituted by using a semiconductor memorysuch as a flash memory or dynamic random access memory (DRAM) andincludes a program recording unit 931 configured to temporarily recordvarious programs executed by the medical observation system 1 and databeing processed.

The second control unit 94 integrally controls the respective units ofthe medical observation system 1. The second control unit 94 is achievedby using a general-purpose processor, such as a CPU having an internalmemory (not illustrated) in which a program is recorded, or a dedicatedprocessor having various arithmetic circuits, such as an ASIC, forperforming a specific function. Furthermore, the second control unit 94may include an FPGA that is a type of a programmable integrated circuit.Note that when the FPGA is included, a memory for storing configurationdata may be provided so that the FPGA as the programmable integratedcircuit is configured based on the configuration data read from thememory.

Processing of Medical Observation System

Next, a process performed by the medical observation system 1 will bedescribed. FIG. 6 is a flowchart illustrating the outline of the processperformed by the medical observation system 1.

As illustrated in FIG. 6, the second control unit 94 determines whetherthe medical observation system 1 is set to a white light observationmode in which white light is emitted to an object (Step S101). If thesecond control unit 94 determines that the medical observation system 1is set to the white light observation mode in which white light isemitted to the object (Step S101: Yes), the medical observation system 1proceeds to Step S102 which is described later. On the other hand, ifthe second control unit 94 determines that the medical observationsystem 1 is not set to the white light observation mode in which whitelight is emitted to the object (Step S101: No), the medical observationsystem 1 proceeds to Step S107 which is described later.

In Step S102, the second control unit 94 causes the first light sourceunit 31, the second light source unit 32, and the third light sourceunit 33 to emit white light.

At this time, the fourth light source unit 34 is turned off.

Next, the second control unit 94 controls the first control unit 25 tocause the imaging unit 21 to receive reflected light from an object tocapture an image (Step S103).

Then, the image processing unit 91 performs various image processing onimage data input from the imaging unit 21 to generate a white lightobservation image (Step S104).

Subsequently, the display device 8 displays the white light observationimage input from the image processing unit 91 (Step S105). This makes itpossible for a user such as a doctor to observe an object to beobserved.

Then, the second control unit 94 determines whether an instructionsignal for terminating the observation of the object to be observed isinput from the input unit 92 (Step S106). If it is determined by thesecond control unit 94 that the instruction signal for terminating theobservation of the object to be observed is input from the input unit 92(Step S106: Yes), the medical observation system 1 finishes thisprocess. On the other hand, if it is determined that no instructionsignal for terminating the observation of the object to be observed isinput from the input unit 92 (Step S106: No), the medical observationsystem 1 returns to Step S101 described above.

In Step S107, the second control unit 94 determines whether the medicalobservation system 1 is set to a fluorescence observation mode foremitting at least excitation light to the object (Step S107). If thesecond control unit 94 determines that the medical observation system 1is set to the fluorescence observation mode for emitting at leastexcitation light to the object (Step S107: Yes), the medical observationsystem 1 proceeds to Step S108 which is described later. On the otherhand, if the second control unit 94 determines that the medicalobservation system 1 is not set to the fluorescence observation mode foremitting at least excitation light to the object (Step S107: No), themedical observation system 1 proceeds to Step S106.

In Step S108, the second control unit 94 causes the fourth light sourceunit 34 to emit excitation light to the object to which a fluorescentsubstance is administered, causing the first light source unit 31 andthe second light source unit 32 to emit light to irradiate the objectwith red light and green light (Step S108). Specifically, as illustratedin FIG. 7, the second control unit 94 causes the fourth light sourceunit 34 in the light source device 3 to emit excitation light IR₁ to theobject O₁ to which the fluorescent substance has been administered. Inthis case, the second control unit 94 causes the first light source unit31 and the second light source unit 32 to emit light at the same time asthe fourth light source unit 34, thereby emitting the excitation lightIR₁, red light W_(R), and green light W_(G) to the object O.Furthermore, the third light source unit 33 is turned off.

Next, the second control unit 94 causes the imaging unit 21 to receivefluorescence IR₂ emitted from the object O₁ to capture an image, andcauses the imaging unit 21 to receive light returned from the object O₁or red light W_(R) and green light W_(G) that are reflected light fromthe object O₁ to capture an image (Step S109). In this case, asillustrated in FIG. 7, the cut filter 213 blocks the excitation lightIR₁ reflected from the object O₁ and transmits the fluorescence IR₂, redlight W_(R), and green light W_(G) from the object O₁. Furthermore, inthe respective pixels (the R pixels, G pixels, and B pixels) in theimage sensor 212, the filters (the filters R, filters G, and filters B)are sensitive to the infrared range. Therefore, the light source device3 emits no red light and the excitation light IR₁ is blocked by the cutfilter 213, and thereby, only the fluorescence IR₂ is incident on the Bpixels of the image sensor 212. Furthermore, the fluorescence IR₂ andthe red light W_(R) that is reflected from the object O₁ are incident onthe R pixels of the image sensor 212. Furthermore, the fluorescence IR₂and the green light W_(G) that is reflected from the object O₁ areincident on the G pixels of the image sensor 212. At this time, in apixel value output from each of the R pixels and the G pixels, becausethe intensity of the fluorescence IR₂ incident on each of the R pixelsand the G pixels is weaker than the intensities of the reflected redlight and the reflected green light, the reflected red light andreflected green light dominate. Furthermore, no blue light is emittedfrom the light source device 3, and thus, the pixel value of each Bpixel is dominated by the fluorescence IR₂. In other words, the imageprocessing unit 91 may use an output value of the B pixel as an outputvalue of the fluorescence IR₂, and output values of the R pixel and theG pixel may be used as output values of visible light (reflected redlight and reflected green light).

Then, the subtraction unit 911 performs a subtraction process forsubtracting the second pixel value from the first or third pixel valueincluded in the image data input from the imaging unit 21 (Step S110).The second pixel value is output from a pixel (B pixel) on which thesecond filter (filter B) is arranged, and the first or third pixel valueis output from a pixel (R pixel or G pixel) on which the first or thirdfilter (filter R or filter G) is arranged.

Here, the subtraction process performed by the subtraction unit 911 willbe described in detail.

When the value of a fluorescent component included in a pixel valueoutput from a Gr pixel (hereinafter, simply referred to as “IRgr”), thevalue of a fluorescent component included in a pixel value output from aGb pixel (hereinafter, simply referred to as “IRgb”), and the value of afluorescent component included in a pixel value output from an R pixel(hereinafter, simply referred to as “IRr”) are obtained, a pixel valuefrom which the fluorescent component of each pixel is removed may beestimated from the value of a fluorescent component included in a pixelvalue output from a B pixel (hereinafter, simply referred to as “IRb”).Specifically, when the spectral sensitivities of the R pixel, G pixel(Gr pixel, Gb pixel), and B pixel on the straight line L_(IR) of FIG. 5are defined as r, g, and b [%], the fluorescent components input to eachof the R pixel, G pixel (Gr pixel, Gb pixel), and B pixel are consideredto be constant. Therefore, the following formulas hold.

IRgr≈IRgb≈(g/b)*IRb   (1)

IRr≈(r/b)*IRb   (2)

Furthermore, the light source device 3 emits no blue light, and thus,the pixel value of the B pixel=IRb. Therefore, when the pixel value ofthe R pixel is R, the pixel value of the Gr pixel is Gr, the pixel valueof the Gb pixel is Gb, and the pixel value of the B pixel is B, thesubtraction unit 911 uses the following formulas (3) to (5) to calculatea value by subtracting the value of the fluorescent component from eachof the pixel value of the R pixel and pixel value of the G pixel.

Pixel value of Gr pixel=Gr−(g/b)*IRgr≈Gr−(g/b)*B   (3)

Pixel value of Gb pixel=Gb−(g/b)*IRgb≈Gb−(g/b)*B   (4)

Pixel value of R pixel=R−(r/b)*IRr≈R−(r/b)*B   (5)

Next, under the control by the second control unit 94, the imageprocessing unit 91 generates a fluorescence image and a first backgroundimage (Step S111). Specifically, the first generation unit 912 uses apixel value of each B pixel included in the image data input from theimage sensor 212 to generate the fluorescence image. In this case, asillustrated in FIG. 7, the first generation unit 912 generates afluorescence image P1 in which the pixel value of each B pixel is usedto interpolate a pixel value corresponding to the position of each ofthe R pixels and G pixels. This fluorescence image P1 is a fluorescenceintensity image. Furthermore, the second generation unit 913 uses thepixel value of each of the R pixels and G pixels included in the imagedata and generates a first background image P2. The image data is inputfrom the image sensor 212 and then input from the subtraction unit 911.In this case, as illustrated in FIG. 7, the second generation unit 913uses the pixel values of each of the R pixels and B pixels to generatethe first background image P2 in which the pixel value corresponding tothe position of each B pixel is interpolated. This first backgroundimage P2 has only a red and a green components and has no blue componentthat has been present in white light observation, and thus, for example,when the pixel value of the B pixel is set to black to generate thefirst background image P2 and then an image is generated on the firstbackground image P2 by demosaic processing, the image has unnaturalcolor. Therefore, the first background image P2 illustrated in FIG. 7 ishatched in order to show an unnatural color.

Then, the image processing unit 91 generates an interpolation imagevalue in which the pixel value of a B pixel is interpolated (Step S112).Specifically, the second generation unit 913 generates the interpolationpixel value based on the first or third pixel value included in theimage data input from the subtraction unit 911. The first or third pixelvalue is output from a pixel (R pixel or G pixel) on which the first orthird filter (filter R or filter G) is arranged, and in theinterpolation pixel value, a pixel value corresponding to a component ofone of red, green, and blue light that is not included in the visiblelight emitted from the light source device 3 is interpolated. Forexample, the second generation unit 913 generates an interpolation pixelvalue in which a pixel value corresponding to the component of bluelight output from each B pixel is interpolated, based on a third pixelvalue output from a G pixel, and outputs the interpolation pixel valueto the third generation unit 914. Note that the second generation unit913 may generate an interpolation pixel value in which a pixel valuecorresponding to the component of blue light output from each B pixel isinterpolated, based on a first pixel value output from an R pixel, andoutputs the interpolation pixel value to the third generation unit 914.

Here, an interpolation method of interpolation by the second generationunit 913 will be described. FIGS. 8A to 8C are each a graph illustratinga correlation between pixels located at the same positions of a redimage, green image, and blue image that are obtained by separating amedical image of a surgical site (biological tissue) of a subject. FIGS.9A to 9C are each a graph illustrating a correlation between pixelslocated at the same positions of a red image, green image, and blueimage that are obtained by separating another medical image of asurgical site (biological tissue) of the subject. FIG. 10 is a tableillustrating a correlation between pixels located at the same positionsof red images, green images, and blue images that are obtained byseparating a plurality of medical images of a surgical site (biologicaltissue) of the subject.

Images of a surgical site captured for medical use show blood, fat, orthe like, and thus, are dominated by colors of red, yellow, and white.Therefore, as illustrated in FIGS. 8A to 8C, 9A to 9C, and Table T1 ofFIG. 10, when the images captured for medical use are separated into ared image, a green image, and a blue image and correlated at the sameposition of the respective images, a correlation between a G pixel and aB pixel is much higher than the other correlations. Therefore, thesecond generation unit 913 generates an interpolation pixel value of a Bpixel by interpolating the pixel value of the B pixel to beinterpolated, from the pixel value of a G pixel located in the vicinityof the B pixel. Specifically, the second generation unit 913 generatesthe interpolation pixel value of a target B pixel (B pixel of interest)by using the following methods 1 to 6 or the like.

Method 1: A value obtained by replicating the pixel value of a Gr pixeladjacent to the target B pixel is used as the interpolation pixel value.

Method 2: A value obtained by replicating the pixel value of a Gb pixeladjacent to the target B pixel is used as the interpolation pixel value.

Method 3: An average value of pixel values of a Gr pixel and a Gb pixelthat are adjacent to the target B pixel is used as the interpolationpixel value.

Method 4: A larger pixel value obtained after comparison between pixelvalues of a Gr pixel and Gb pixel adjacent to the target B pixel is usedas the interpolation pixel value.

Method 5: A smaller pixel value obtained after comparison between pixelvalues of a Gr pixel and Gb pixel adjacent to the target B pixel is usedas the interpolation pixel value.

Method 6: An average value of pixel values of a plurality of Gr pixelsand Gb pixels located in a predetermined range (e.g., 3×3 pixels) aroundthe target B pixel is used as the interpolation pixel value.

Here, a generation method for generating the fluorescence image P1 bythe first generation unit 912 will be described. FIG. 11 is a diagramschematically illustrating the generation method for generating thefluorescence image P1 by the first generation unit 912.

As illustrated in FIG. 11, the first generation unit 912 interpolates apixel value corresponding to the position of each of an R pixel and Gpixels by replicating the pixel value of the B pixel (pixel P_(n,m)) asthe pixel values of the R pixel (A3) and G pixels (A1, A2), withoutusing the pixel value of each of the R pixel (A3) and G pixels (A1, A2)in a unit Z1 in a Bayer array, which is included in image data inputfrom the image sensor 212. Likewise, in a unit Z2, the first generationunit 912 replicates the pixel value of a B pixel (pixel P_(n,m+2)) asthe pixel values of an R pixel and G pixel (e.g., A4) to interpolate thepixel value corresponding to the position of each of the R pixel and Gpixel. Furthermore, in a unit Z3, the first generation unit 912replicates the pixel value of a B pixel (pixel P_(n+2,m)) as the pixelvalue of an R pixel and G pixel (e.g., A5) to interpolate the pixelvalue corresponding to the position of each of the R pixel and G pixel.In this way, the first generation unit 912 uses the pixel value of a Bpixel in each unit to generate the fluorescence image P1 in which thepixel value corresponding to the position of each of the R pixel and Gpixel is interpolated.

FIG. 12 is a schematic diagram illustrating another example of thegeneration method for generating the fluorescence image P1 by the firstgeneration unit 912.

As illustrated in FIG. 12, the first generation unit 912 uses an averagevalue of the pixel values of adjacent B pixels to interpolate the pixelvalue corresponding to the position of each of the R pixel and the Gpixels, and generates the fluorescence image P1. Specifically, the firstgeneration unit 912 sets the pixel value of a G pixel (A1) to an averagevalue ((pixel value of pixel P_(n,m))+(pixel value of pixelP_(n,m+2))/2) of the pixel value of a B pixel (pixel P_(n,m)) and thepixel value of a B pixel (pixel P_(n,m+2)) that are adjacent to the Gpixel. Likewise, the first generation unit 912 sets the pixel value of aG pixel (A2) to an average value ((pixel value of pixelP_(n+2,m))+(pixel value of pixel P_(n,m))/2) of the pixel value of a Bpixel (pixel P_(n,m)) and the pixel value of a B pixel (pixel P_(n+2,m))that are adjacent to the G pixel. Furthermore, the first generation unit912 sets the pixel value of an R pixel (A3) to an average value ((pixelvalue of pixel P_(n2,m))+(pixel value of pixel P_(n+2,m+2))/2) of thepixel value of a B pixel (pixel P_(n,m)) and the pixel value of a Bpixel (pixel P_(n+2,m+2)) that are adjacent to the R pixel. In this way,the first generation unit 912 uses an average value of the pixel valuesof adjacent B pixels to interpolate the pixel value corresponding to theposition of each of the R pixel and G pixels, and generates thefluorescence image P1.

FIG. 13 is a diagram schematically illustrating a generation method forgenerating the first background image P2 by the second generation unit913.

As illustrated in FIG. 13, the second generation unit 913 considers a Bpixel (pixel P_(n,m)) as black color (pixel value is 0) and generatesthe first background image P2.

Returning to FIG. 6, the description will be continued from Step S113.

In Step S113, the third generation unit 914 generates a secondbackground image based on the interpolation pixel value and the firstbackground image that are input from the second generation unit 913.Specifically, as illustrated in FIG. 7, the third generation unit 914combines the interpolation pixel value and the first background image P2that are input from the second generation unit 913, performs demosaicprocessing, and generates a second background image P3. Thus, the secondbackground image P3 has a color closer to the color in white lightobservation.

Subsequently, the third generation unit 914 may perform binarization forthe second background image P3 and output the second background image P3to the combining unit 915 (Step S114). Specifically, the thirdgeneration unit 914 may perform grayscale processing on the secondbackground image P3 to generate a grayscale image and outputs thegrayscale image to the combining unit 915.

Then, the first generation unit 912 performs colorization for thefluorescence image P1 and outputs the fluorescence image P1 to thecombining unit 915 (Step S115). Specifically, the first generation unit912 colorizes the fluorescence image P1 by a tone conversion process orthe like based on the brightness value of the fluorescence image P1, andoutputs the colored fluorescence image P1 to the combining unit 915. Forexample, the first generation unit 912 colorizes the fluorescence imageP1 green.

Then, the combining unit 915 generates a composite image P4 in which thefluorescence image P1 generated by the first generation unit 912 and thesecond background image P3 generated by the third generation unit 914are combined (Step S116). Specifically, as illustrated in FIG. 7, thecombining unit 915 generates the composite image P4 in which thefluorescence image P1 and the second background image P3 (backgroundimage) are combined, and outputs the composite image P4 to the displaydevice 8. In this case, the combining unit 915 generates the compositeimage P4 by combining the fluorescence image P1 with the secondbackground image P3 at a predetermined ratio (e.g., 1:1).

Then, the display device 8 displays the composite image P4 input fromthe combining unit 915 (Step S117). Thus, as illustrated in FIG. 7, theuser such as a doctor may grasp the position of a fluorescent area Q1 byobserving the composite image P4 of a color closer to natural color thatis displayed on the display device 8. After Step S117, the medicalobservation system 1 proceeds to Step S106.

According to the first embodiment described above, the image processingunit 91 generates the interpolation pixel value in which a pixel valuecorresponding to the component of blue light is interpolated, based onthe first or third pixel value (R pixel or G pixel), generates thebackground image (second background image) corresponding to eachcomponent of light in red, green, and blue wavelength bands, based onthe first or third pixel value and the interpolation pixel value, andgenerates the fluorescence image, based on the second pixel value. Thefirst or third pixel value is included in image data and output from apixel on which the first or third filter (filter R or filter G) isarranged, and the second pixel value is included in the image data andoutput from a pixel (B pixel) on which the second filter (filter B) isarranged. In this method, visible light and infrared excitation lightare preferably emitted from the light source device 3 at the same time,and it is not necessary to alternately emit the visible light andinfrared excitation light, and thus flickering of an observed region ofan object to be observed may be prevented and the size of the device maybe reduced.

Furthermore, according to the first embodiment, the fluorescence imageP1 and the normal white light observation image are allowed to begenerated by using one image sensor 212 having a normal Bayer array, andthus observation of the observed region of the object to be observed ispossible while appropriately switching between the white lightobservation mode and the fluorescence observation mode, without using aspecial image sensor.

Furthermore, in the first embodiment, the image processing unit 91generates the pixel value of a B pixel as the interpolation pixel value,based on the pixel value of a G pixel, and thus it becomes possible togenerate the background image (second background image) having a colorcloser to that of the white light observation.

Furthermore, according to the first embodiment, after binarization forthe background image (second background image), the image processingunit 91 combines the binarized background image (second backgroundimage) with the fluorescence image, and thereby it is possible toemphasize the fluorescent area on the composite image.

Furthermore, according to the first embodiment, after colorization ofthe fluorescence image, the image processing unit 91 combines thecolorized fluorescence image with the background image (secondbackground image), and thus, it is possible to emphasize the fluorescentarea on the composite image.

Furthermore, according to the first embodiment, when the medicalobservation system 1 is set to the white light observation mode, thesecond control unit 94 causes the first light source unit 31, the secondlight source unit 32, and the third light source unit 33 to emit whitelight, and when the medical observation system 1 is set to thefluorescence observation mode, the second control unit 94 causes thefirst light source unit 31, the second light source unit 32, and thefourth light source unit 34 to simultaneously emit visible light andexcitation light, and thus, it is possible to observe the observedregion of the object to be observed while appropriately switchingbetween the white light observation mode and the fluorescenceobservation mode.

Furthermore, according to the first embodiment, the image processingunit 91 divides the spectral sensitivity of the first filter (filter R,filter G) by the spectral sensitivity of the second filter (filter B) toobtain a divided value, multiplies the divided value by the second pixelvalue (pixel value of the B pixel) to obtain a multiplication result,subtracts the multiplication result from the first pixel value (pixelvalue of each of the R pixel and G pixel) to obtain a subtractionresult, and generates the background image based on the subtractionresult. Therefore, it is possible to generate the background image fromwhich the fluorescent component is removed.

Note that in the first embodiment, the image processing unit 91 dividesthe spectral sensitivity of a fluorescence wavelength of the firstfilter (filter R, filter G) by the spectral sensitivity of afluorescence wavelength of the second filter (filter B) to obtain adivided value, multiplies the divided value by the second pixel value(pixel value of the B pixel) to obtain a multiplication result, andsubtracts the multiplication result from the first pixel value (pixelvalue of each of the R pixel and G pixel), but the second pixel valuemay merely be subtracted from the first pixel value.

Furthermore, in the first embodiment, when the medical observationsystem 1 is set to the fluorescence observation mode, the second controlunit 94 causes each of the second light source unit 32 and the thirdlight source unit 33 to emit light, but the second control unit 94 maycause the first light source unit 31 and one of the second light sourceunit 32 and the third light source unit 33 to emit visible light (redlight+green light or red light+blue light). For example, when causingthe first light source unit 31 and the third light source unit 33 toemit illumination light (red light+blue light) to the object, the imageprocessing unit 91 preferably generates the background image and thefluorescence image by performing the processing similar to theprocessing for the G pixel as described above, for the B pixel asdescribed above, and the processing similar to the processing for the Bpixel as described above, for the G pixel.

Furthermore, in the first embodiment, the image processing unit 91performs binarization for the background image, but may performbinarization for the fluorescence image, or may perform binarization foreach of the background image and the fluorescence image. This makes itpossible to emphasize the fluorescent area on the composite image. As amatter of course, the image processing unit 91 may omit the binarizationfor each of the background image and the fluorescence image. This makesit possible to simplify the process.

Furthermore, in the first embodiment, the image processing unit 91performs colorization for the fluorescence image, but may performcolorization for the background image, or may perform colorization forthe background image and the fluorescence image. In this case, the imageprocessing unit 91 performs colorization for each of the backgroundimage and the fluorescence image so that the background image and thefluorescence image have different colors. This makes it possible toemphasize the fluorescent area on the composite image. As a matter ofcourse, the image processing unit 91 may omit the colorization for eachof the background image and the fluorescence image. This makes itpossible to simplify the process.

Furthermore, in the first embodiment, the first light source unit 31,the second light source unit 32, and the fourth light source unit 34 arecaused to emit light in the fluorescence observation mode, but, forexample, a light source configured to be able to emit white light and acut filter configured to block a blue wavelength band and transmit lightin a wavelength band other than the blue wavelength band may be providedso that the cut filter is arranged on a white-light optical path throughwhich white light is emitted, in the fluorescence observation mode. As amatter of course, the cut filter having a transmission characteristic ofblocking a green wavelength band and transmitting light in a wavelengthband other than the green wavelength band may be applied.

Furthermore, in the first embodiment, a cut filter configured to blockone of the green and blue wavelength bands and transmit light in awavelength band other than the green and blue wavelength bands may beemployed, even if the cut filter is removably provided on the opticalpath between the optical system 211 and the cut filter 213 so as to beinserted in the optical path between the optical system 211 and the cutfilter 213, in the fluorescence observation mode.

Modification of First Embodiment

Next, a modification of the first embodiment will be described. In thefirst embodiment described above, an image is captured by one imagesensor 212 (single plate) having the Bayer array, but in themodification of the first embodiment, a plurality of image sensors isused.

FIG. 14 is a conceptual diagram illustrating a configuration of animaging unit according to a modification of the first embodiment. Theimaging unit 21A illustrated in FIG. 14 includes an image sensor 212R,an image sensor 212G, an image sensor 212B, and a dichroic prism 214, inplace of the image sensor 212 described above.

The image sensor 212R includes the pixel portion 212 a described above,receives red light split by the dichroic prism 214 which is describedlater, and performs photoelectric conversion to generate image data.

The image sensor 212G includes the pixel portion 212 a described above,receives green light split by the dichroic prism 214 which is describedlater, and performs photoelectric conversion to generate image data.

The image sensor 212B includes the pixel portion 212 a described above,receives blue light split by the dichroic prism 214 which is describedlater, and performs photoelectric conversion to generate image data.

The dichroic prism 214 emits red light and fluorescence, obtained fromlight incident thereon through the cut filter 213, to the image sensor212R, emits green light and fluorescence to the image sensor 212G, andemits blue light and fluorescence to the image sensor 212B.

The medical observation system including the imaging unit 21A havingsuch a configuration performs processing similar to that in the medicalobservation system 1 described above (see FIG. 6).

According to the modification of the first embodiment described above,the effects similar to those of the first embodiment described above maybe obtained.

Second Embodiment

Next, a second embodiment will be described. In the first embodimentdescribed above, the surgical microscope has been described as themedical observation system, but in the second embodiment, an endoscopesystem having a rigid endoscope will be described as the medicalobservation system. Note that the same configurations as those of themedical observation system 1 according to the first embodiment describedabove are denoted by the same reference signs, and detailed descriptionthereof will be omitted.

Configuration of Medical Observation System

FIG. 15 is a diagram illustrating a schematic configuration of themedical observation system according to the second embodiment. A medicalobservation system 1B illustrated in FIG. 15 is used in the medicalfield and is a system configured to observe biological tissue in asubject such as a living body. Note that in the second embodiment, arigid endoscope system using the rigid endoscope (insertion section 102)illustrated in FIG. 15 will be described as the medical observationsystem 1B.

The medical observation system 1B illustrated in FIG. 15 includes theinsertion section 102, a light source device 3, a light guide 104, anendoscope camera head 105 (endoscopic imaging device), a firsttransmission cable 106, a display device 8, a second transmission cable108, a control device 9, and a third transmission cable 1010.

The insertion section 102 is rigid or at least partially flexible andhas an elongated shape. The insertion section 102 is inserted into thesubject such as a patient through a trocar. The insertion section 102 isinternally provided with an optical system, such as a lens, configuredto form an observation image.

The light guide 104 has one end that is detachably connected to thelight source device 3 and the other end that is detachably connected tothe insertion section 102. The light guide 104 guides illumination lightsupplied from the light source device 3 from the one end to the otherend and supplies the illumination light to the insertion section 102.

The insertion section 102 includes an eyepiece 121 that is detachablyconnected to the endoscope camera head 105. Under the control by thecontrol device 9, the endoscope camera head 105 receives light focusedinto an observation image by the insertion section 102, performsphotoelectric conversion to generate image data (RAW data), and outputsthe image data to the control device 9 via the first transmission cable106.

The first transmission cable 106 has one end that is detachablyconnected to the control device 9 via a video connector 161, and theother end that is detachably connected to the endoscope camera head 105via a camera head connector 162. The first transmission cable 106transmits image data output from the endoscope camera head 105 to thecontrol device 9 and transmits setting data, power, or the like outputfrom the control device 9 to the endoscope camera head 105.

The second transmission cable 108 has one end that is detachablyconnected to the display device 8, and the other end that is detachablyconnected to the control device 9. The second transmission cable 108transmits image data processed by the control device 9 to the displaydevice 8.

The third transmission cable 1010 has one end that is detachablyconnected to the light source device 3, and the other end that isdetachably connected to the control device 9. The third transmissioncable 1010 transmits control data from the control device 9 to the lightsource device 3.

Functional Configuration of Main Portion of Medical Observation System

Next, the functional configuration of a main portion of the medicalobservation system 1B described above will be described. FIG. 16 is ablock diagram illustrating a functional configuration of the mainportion of the medical observation system 1B.

Configuration of Endoscope Camera Head

First, a configuration of the endoscope camera head 105 will bedescribed. The endoscope camera head 105 includes an image sensor 212, acut filter 213, a lens unit 501, a camera head memory 502, and a camerahead controller 503.

The lens unit 501 forms an image of an object focused by the opticalsystem of the insertion section 102 on a light receiving surface of theimage sensor 212. The focal position of the lens unit 501 is changeable.The lens unit 501 includes a plurality of lenses.

The camera head memory 502 records various information about theendoscope camera head 105 (e.g., pixel information about the imagesensor 212, characteristics of the cut filter 213). Furthermore, thecamera head memory 502 records various setting data and controlparameters transmitted from the control device 9 via the firsttransmission cable 106. The camera head memory 502 includes anon-volatile memory or a volatile memory.

The camera head controller 503 controls the operation of each unitconstituting the endoscope camera head 105 based on the setting datareceived from the control device 9 via the first transmission cable 106.The camera head controller 503 includes a timing generator (TG), aprocessor that is a processing device having hardware such as CPU, and amemory that is a temporary storage area used by the processor.

The medical observation system 1B having such a configuration performsprocessing similar to that in the medical observation system 1 describedabove (see FIG. 6).

According to the second embodiment described above, the effects similarto those of the first embodiment described above may be obtained,reducing the size of the endoscope camera head 105.

Third Embodiment

Next, a third embodiment will be described. In the third embodiment, amedical observation system that is applied to a flexible endoscopesystem using a flexible endoscope will be described. Note that the sameconfigurations as those of the medical observation system 1 according tothe first embodiment described above are denoted by the same referencesigns, and detailed description thereof will be omitted.

Schematic Configuration of Medical Observation System

FIG. 17 is a diagram illustrating a schematic configuration of themedical observation system according to the third embodiment. A medicalobservation system 1C illustrated in FIG. 17 is configured to beinserted into a subject, image inside the subject to generate imagedata, and display an image based on the image data.

As illustrated in FIG. 17, the medical observation system 1C includes anendoscope 201 configured to capture an in-vivo image of an observedregion by inserting an insertion section 202 into the subject togenerate image data, a light source device 3, a display device 8, and acontrol device 9. The endoscope 201 is provided with an imaging unit 21at a distal end portion 203 of the insertion section 202.

The medical observation system 1C having such a configuration performsprocessing similar to that in the medical observation system 1 describedabove (see FIG. 6).

According to the third embodiment described above, even the medicalobservation system 1C including the flexible endoscope 201 may obtainthe effects similar to those of the first embodiment described above.

Other Embodiments

Various aspects of the disclosure may be formed by appropriatelycombining a plurality of component elements disclosed in the medicalobservation system according to the first to third embodiments of thepresent disclosure described above. For example, some component elementsmay be deleted from all the component elements described in the medicalobservation system according to the first to third embodiments of thepresent disclosure described above. Furthermore, the component elementsdescribed in the medical observation system according to the embodimentsof the present disclosure described above may be appropriately combined.

Furthermore, in the medical observation system according to the first tothird embodiments of the present disclosure, the word “unit” describedabove may be read as “means”, “circuit”, or the like. For example, thecontrol unit may be read as control means or a control circuit.

Furthermore, a program executed by the medical observation systemaccording to the first to third embodiments of the present disclosure isprovided in the form of installable or executable file data and recordedin a computer-readable recording medium, such as a CD-ROM, flexible disk(FD), CD-R, digital versatile disk (DVD), USB medium, or flash memory.

Alternatively, a program executed by the medical observation systemaccording to the first to third embodiments of the present disclosuremay be configured to be stored on a computer connected to a network suchas the Internet and provided by being downloaded via the network.

It is noted that, in the description of the flowchart herein, a contextof processes between timings has been clearly shown by usingexpressions, such as “first”, “then”, and “subsequently”, but the orderof processes necessary to carry out the present disclosure is notuniquely defined by these expressions. In other words, the order of theprocesses in the flowchart described herein may be changed or modifiedwithin a consistent range. For example, the generation and colorizationof the fluorescence image and the generation and binarization of thebackground image may be performed in parallel.

Some embodiments of the present application have been described indetail with reference to the drawings, but these are provided by way ofexamples, and it is possible to carry out the present disclosure inother forms, including the modes described in the present disclosure, towhich various modifications and improvements may be made based on theknowledge of those skilled in the art.

Note that the present technique may also have the followingconfigurations.

(Supplementary Note 1)

A medical image processing device including

an image processing unit configured to perform image processing, basedon image data generated by imaging reflected light of first visiblelight emitted to an object and fluorescence, by a medical imagingdevice, when a light source device simultaneously emits, to the object,the first visible light and excitation light that excites a fluorescentsubstance to emit the fluorescence,

wherein the image processing unit is configured to

generate an interpolation pixel value corresponding to a component ofsecond visible light in a band different from a wavelength band of thefirst visible light, based on a first pixel value included in the imagedata and output from a pixel receiving the reflected light of the firstvisible light emitted to the object,

generate a background image based on the first pixel value and theinterpolation pixel value, and

generate a fluorescence image based on a second pixel value included inthe image data and output from a pixel receiving the fluorescence.

(Supplementary Note 2)

The medical image processing device according to Supplementary note 1,wherein the medical imaging device includes a cut filter provided nearan incident surface of an image sensor and configured to block theexcitation light but transmit the reflected light of the first visiblelight emitted to the object and the fluorescence.

(Supplementary Note 3)

The medical image processing device according to Supplementary note 1 or2, wherein the image processing unit is configured to:

subtract the second pixel value from the first pixel value; and

generate the background image, based on a result of the subtraction andthe interpolation pixel value.

(Supplementary Note 4)

The medical image processing device according to Supplementary note 1,wherein

the medical imaging device includes an image sensor including

-   -   a plurality of pixels,    -   a first filter configured to transmit the first visible light        and the fluorescence, and a second filter configured to transmit        the second visible light and the fluorescence, the first filter        and the second filter each being provided on a light receiving        surface of each of the plurality of pixels,

the image sensor is configured to

-   -   image at least one of reflected light of at least one of the        first visible light and the second visible light that are        emitted to the object and the fluorescence, and    -   generate image data,

the first pixel value is output from a pixel on which the first filteris arranged, and

the second pixel value is output from a pixel on which the second filteris arranged.

(Supplementary Note 5)

The medical image processing device according to Supplementary note 4,wherein the image processing unit is configured to:

divide a spectral sensitivity of a fluorescence wavelength of the firstfilter by a spectral sensitivity of a fluorescence wavelength of thesecond filter to obtain a divided value,

multiply the divided value by the second pixel value to obtain amultiplication result,

subtract the multiplication result from the first pixel value to obtaina subtraction result, and

generate the background image based on the subtraction result and theinterpolation pixel value.

(Supplementary Note 6)

The medical image processing device according to Supplementary note 4 or5, wherein

the first visible light is one of light in a red wavelength band andlight in a green wavelength band,

the first filter is one of a red filter configured to transmit the lightin a red wavelength band and the fluorescence and a green filterconfigured to transmit the light in a green wavelength band and thefluorescence, and

the second filter is a blue filter configured to transmit light in ablue wavelength band and the fluorescence.

(Supplementary Note 7)

The medical image processing device according to Supplementary note 6,wherein

the light source device is configured to emit third visible light havinga wavelength band different from the wavelength bands of the firstvisible light and the second visible light,

the image sensor includes a third filter configured to transmit thethird visible light and the fluorescence,

the third visible light is another one of the light in a red wavelengthband and the light in a green wavelength band, and

the first filter is another one of the red filter configured to transmitthe light in a red wavelength band and the fluorescence and the greenfilter configured to transmit the light in a green wavelength band andthe fluorescence.

(Supplementary Note 8)

The medical image processing device according to Supplementary note 7,wherein the image processing unit is configured to generate theinterpolation pixel value based on the first pixel value output from apixel on which the green filter is arranged.

(Supplementary Note 9)

The medical image processing device according to Supplementary note 4 or5, wherein

the first visible light is one of light in a red wavelength band andlight in a blue wavelength band,

the first filter is one of a red filter configured to transmit the lightin a red wavelength band and the fluorescence and a blue filterconfigured to transmit the light in a blue wavelength band and thefluorescence, and

the second filter is a green filter configured to transmit light in agreen wavelength band and the fluorescence.

(Supplementary Note 10)

The medical image processing device according to Supplementary note 9,wherein

the light source device is configured to emit third visible light havinga wavelength band different from the wavelength bands of the firstvisible light and the second visible light,

the image sensor includes a third filter configured to transmit thethird visible light and the fluorescence,

the third visible light is another one of the light in a red wavelengthband and the light in a blue wavelength band, and

the first filter is another one of the red filter configured to transmitthe light in a red wavelength band and the fluorescence and the bluefilter configured to transmit the light in a blue wavelength band andthe fluorescence.

(Supplementary Note 11)

The medical image processing device according to Supplementary note 10,wherein the image processing unit is configured to generate theinterpolation pixel value based on the first pixel value output from apixel on which the blue filter is arranged.

(Supplementary Note 12)

The medical image processing device according to any one ofSupplementary notes 6 to 11, further including

a control unit configured to control the light source device,

the light source device includes:

-   -   a first light source unit configured to emit light in a red        wavelength band;    -   a second light source unit configured to emit the light in a        green wavelength band;    -   a third light source unit configured to emit the light in a blue        wavelength band; and    -   a fourth light source unit configured to emit the excitation        light,

wherein the control unit is configured to:

-   -   cause, in a white light observation mode for observation with        white light, the first light source unit, the second light        source unit, and the third light source unit to emit light, and    -   cause, in a fluorescence observation mode for observation with        the fluorescence, the first light source unit, the fourth light        source unit, and one of the second light source unit and the        third light source unit to emit light.

(Supplementary Note 13)

The medical image processing device according to any one ofSupplementary notes 1 to 12, wherein the image processing unit isconfigured to generate a composite image obtained by combining thebackground image and the fluorescence image.

(Supplementary Note 14)

The medical image processing device according to any one ofSupplementary notes 1 to 12, wherein the image processing unit isconfigured to perform binarization for at least one of the backgroundimage and the fluorescence image.

(Supplementary Note 15)

The medical image processing device according to any one ofSupplementary notes 1 to 12, wherein the image processing unit isconfigured to perform colorization for at least one of the backgroundimage and the fluorescence image.

(Supplementary Note 16)

The medical image processing device according to any one ofSupplementary notes 1 to 15, wherein

the fluorescent substance employs indocyanine green, and

the excitation light has a center wavelength of 740 nm.

(Supplementary Note 17)

A medical image processing device including

an image processing unit configured to perform image processing, basedon image data generated by imaging reflected light of first visiblelight emitted to an object and fluorescence, by a medical imagingdevice, when a light source device simultaneously emits, to the object,the first visible light and excitation light that excites a fluorescentsubstance to emit the fluorescence,

wherein the medical imaging device includes

-   -   a dichroic prism configured to split the reflected light and the        fluorescence into a plurality of wavelength bands, and    -   a plurality of image sensors configured to receive light split        into the plurality of wavelength bands by the dichroic prism and        generate a plurality of pieces of the image data, and

the image processing unit is configured to:

-   -   generate an interpolation pixel value that interpolates a pixel        value corresponding to a component of second visible light in a        band different from a wavelength band of the first visible        light, based on the image data;    -   generate a background image, based on the image data and the        interpolation pixel value; and    -   generate a fluorescence image, based on the image data.

(Supplementary Note 18)

A medical observation system including:

the medical image processing device according to any one ofSupplementary notes 1 to 17;

a support unit configured to rotatably support the medical imagingdevice; and

a base portion configured to rotatably hold a base end portion of thesupport unit and to be movable on a floor surface.

(Supplementary Note 19)

A medical observation system including:

the medical image processing device according to any one ofSupplementary notes 1 to 17; and

an insertion section configured to be insertable into a subject andincluding an optical system configured to focus the reflected light andthe fluorescence to form an image of an object on a light receivingsurface of an image sensor.

(Supplementary Note 20)

The medical observation system according to Supplementary note 19,wherein

the insertion section is configured to be removable from the medicalimaging device.

(Supplementary Note 21)

A method of processing an image executed by a medical image processingdevice configured to perform image processing, based on image datagenerated by imaging reflected light of first visible light emitted toan object and fluorescence, by a medical imaging device, when a lightsource device simultaneously emits, to the object, the first visiblelight and excitation light that excites a fluorescent substance to emitthe fluorescence, the method including:

generating an interpolation pixel value corresponding to a component ofsecond visible light in a band different from a wavelength band of thefirst visible light, based on a first pixel value included in the imagedata and output from a pixel receiving the reflected light of the firstvisible light emitted to the object;

generating a background image based on the first pixel value and theinterpolation pixel value; and

generating a fluorescence image based on a second pixel value includedin the image data and output from a pixel receiving the fluorescence.

According to the present disclosure, the size of the device may beeffectively reduced.

Although the disclosure has been described with respect to specificembodiments for a complete and clear disclosure, the appended claims arenot to be thus limited but are to be construed as embodying allmodifications and alternative constructions that may occur to oneskilled in the art that fairly fall within the basic teaching herein setforth.

What is claimed is:
 1. A medical image processing device comprising animage processing unit configured to perform image processing, based onimage data generated by imaging reflected light of first visible lightemitted to an object and fluorescence, by a medical imaging device, whena light source device simultaneously emits, to the object, the firstvisible light and excitation light that excites a fluorescent substanceto emit the fluorescence, wherein the image processing unit isconfigured to generate an interpolation pixel value corresponding to acomponent of second visible light in a wavelength band different from awavelength band of the first visible light, based on a first pixel valueincluded in the image data and output from a pixel receiving thereflected light of the first visible light emitted to the object,generate a background image based on the first pixel value and theinterpolation pixel value, and generate a fluorescence image based on asecond pixel value included in the image data and output from a pixelreceiving the fluorescence.
 2. The medical image processing deviceaccording to claim 1, wherein the medical imaging device includes a cutfilter provided near an incident surface of an image sensor andconfigured to block the excitation light but transmit the reflectedlight of the first visible light emitted to the object and thefluorescence.
 3. The medical image processing device according to claim1, wherein the image processing unit is configured to: subtract thesecond pixel value from the first pixel value; and generate thebackground image, based on a result of the subtraction and theinterpolation pixel value.
 4. The medical image processing deviceaccording to claim 1, wherein the medical imaging device includes animage sensor including a plurality of pixels, a first filter configuredto transmit the first visible light and the fluorescence, and a secondfilter configured to transmit the second visible light and thefluorescence, the first filter and the second filter each being providedon a light receiving surface of each of the plurality of pixels, theimage sensor is configured to image at least one of reflected light ofat least one of the first visible light and the second visible lightthat are emitted to the object and the fluorescence, and generate imagedata, the first pixel value is output from a pixel on which the firstfilter is arranged, and the second pixel value is output from a pixel onwhich the second filter is arranged.
 5. The medical image processingdevice according to claim 4, wherein the image processing unit isconfigured to: divide a spectral sensitivity of a fluorescencewavelength of the first filter by a spectral sensitivity of afluorescence wavelength of the second filter to obtain a divided value,multiply the divided value by the second pixel value to obtain amultiplication result, subtract the multiplication result from the firstpixel value to obtain a subtraction result, and generate the backgroundimage based on the subtraction result and the interpolation pixel value.6. The medical image processing device according to claim 4, wherein thefirst visible light is one of light in a red wavelength band and lightin a green wavelength band, the first filter is one of a red filterconfigured to transmit the light in a red wavelength band and thefluorescence and a green filter configured to transmit the light in agreen wavelength band and the fluorescence, and the second filter is ablue filter configured to transmit light in a blue wavelength band andthe fluorescence.
 7. The medical image processing device according toclaim 6, wherein the light source device is configured to emit thirdvisible light having a wavelength band different from the wavelengthbands of the first visible light and the second visible light, the imagesensor includes a third filter configured to transmit the third visiblelight and the fluorescence, the third visible light is another one ofthe light in a red wavelength band and the light in a green wavelengthband, and the first filter is another one of the red filter configuredto transmit the light in a red wavelength band and the fluorescence andthe green filter configured to transmit the light in a green wavelengthband and the fluorescence.
 8. The medical image processing deviceaccording to claim 7, wherein the image processing unit is configured togenerate the interpolation pixel value based on the first pixel valueoutput from a pixel on which the green filter is arranged.
 9. Themedical image processing device according to claim 4, wherein the firstvisible light is one of light in a red wavelength band and light in ablue wavelength band, the first filter is one of a red filter configuredto transmit the light in a red wavelength band and the fluorescence anda blue filter configured to transmit the light in a blue wavelength bandand the fluorescence, and the second filter is a green filter configuredto transmit light in a green wavelength band and the fluorescence. 10.The medical image processing device according to claim 9, wherein thelight source device is configured to emit third visible light having awavelength band different from the wavelength bands of the first visiblelight and the second visible light, the image sensor includes a thirdfilter configured to transmit the third visible light and thefluorescence, the third visible light is another one of the light in ared wavelength band and the light in a blue wavelength band, and thefirst filter is another one of the red filter configured to transmit thelight in a red wavelength band and the fluorescence and the blue filterconfigured to transmit the light in a blue wavelength band and thefluorescence.
 11. The medical image processing device according to claim10, wherein the image processing unit is configured to generate theinterpolation pixel value based on the first pixel value output from apixel on which the blue filter is arranged.
 12. The medical imageprocessing device according to claim 6, further comprising a controlunit configured to control the light source device, the light sourcedevice includes: a first light source unit configured to emit light in ared wavelength band; a second light source unit configured to emit thelight in a green wavelength band; a third light source unit configuredto emit the light in a blue wavelength band; and a fourth light sourceunit configured to emit the excitation light, wherein the control unitis configured to: cause, in a white light observation mode forobservation with white light, the first light source unit, the secondlight source unit, and the third light source unit to emit light, andcause, in a fluorescence observation mode for observation with thefluorescence, the first light source unit, the fourth light source unit,and one of the second light source unit and the third light source unitto emit light.
 13. The medical image processing device according toclaim 1, wherein the image processing unit is configured to generate acomposite image obtained by combining the background image and thefluorescence image.
 14. The medical image processing device according toclaim 1, wherein the image processing unit is configured to performbinarization for at least one of the background image and thefluorescence image.
 15. The medical image processing device according toclaim 1, wherein the image processing unit is configured to performcolorization for at least one of the background image and thefluorescence image.
 16. The medical image processing device according toclaim 1, wherein the fluorescent substance employs indocyanine green,and the excitation light has a center wavelength of 740 nm.
 17. Amedical image processing device comprising an image processing unitconfigured to perform image processing, based on image data generated byimaging reflected light of first visible light emitted to an object andfluorescence, by a medical imaging device, when a light source devicesimultaneously emits, to the object, the first visible light andexcitation light that excites a fluorescent substance to emit thefluorescence, wherein the medical imaging device includes a dichroicprism configured to split the reflected light and the fluorescence intoa plurality of wavelength bands, and a plurality of image sensorsconfigured to receive light split into the plurality of wavelength bandsby the dichroic prism and generate a plurality of pieces of the imagedata, and the image processing unit is configured to: generate aninterpolation pixel value that interpolates a pixel value correspondingto a component of second visible light in a band different from awavelength band of the first visible light, based on the image data;generate a background image, based on the image data and theinterpolation pixel value; and generate a fluorescence image, based onthe image data.
 18. A medical observation system comprising: the medicalimage processing device according to claim 1; a support unit configuredto rotatably support the medical imaging device; and a base portionconfigured to rotatably hold a base end portion of the support unit andto be movable on a floor surface.
 19. A medical observation systemcomprising: the medical image processing device according to claim 1;and an insertion section configured to be insertable into a subject andincluding an optical system configured to focus the reflected light andthe fluorescence to form an image of an object on a light receivingsurface of an image sensor.
 20. A method of processing an image executedby a medical image processing device configured to perform imageprocessing, based on image data generated by imaging reflected light offirst visible light emitted to an object and fluorescence, by a medicalimaging device, when a light source device simultaneously emits, to theobject, the first visible light and excitation light that excites afluorescent substance to emit the fluorescence, the method comprising:generating an interpolation pixel value corresponding to a component ofsecond visible light in a band different from a wavelength band of thefirst visible light, based on a first pixel value included in the imagedata and output from a pixel receiving the reflected light of the firstvisible light emitted to the object; generating a background image basedon the first pixel value and the interpolation pixel value; andgenerating a fluorescence image based on a second pixel value includedin the image data and output from a pixel receiving the fluorescence.